93 research outputs found
Expectation Propagation for Approximate Inference: Free Probability Framework
We study asymptotic properties of expectation propagation (EP) -- a method
for approximate inference originally developed in the field of machine
learning. Applied to generalized linear models, EP iteratively computes a
multivariate Gaussian approximation to the exact posterior distribution. The
computational complexity of the repeated update of covariance matrices severely
limits the application of EP to large problem sizes. In this study, we present
a rigorous analysis by means of free probability theory that allows us to
overcome this computational bottleneck if specific data matrices in the problem
fulfill certain properties of asymptotic freeness. We demonstrate the relevance
of our approach on the gene selection problem of a microarray dataset.Comment: Both authors are co-first authors. The main body of this paper is
accepted for publication in the proceedings of the 2018 IEEE International
Symposium on Information Theory (ISIT
A Theory of Solving TAP Equations for Ising Models with General Invariant Random Matrices
We consider the problem of solving TAP mean field equations by iteration for
Ising model with coupling matrices that are drawn at random from general
invariant ensembles. We develop an analysis of iterative algorithms using a
dynamical functional approach that in the thermodynamic limit yields an
effective dynamics of a single variable trajectory. Our main novel contribution
is the expression for the implicit memory term of the dynamics for general
invariant ensembles. By subtracting these terms, that depend on magnetizations
at previous time steps, the implicit memory terms cancel making the iteration
dependent on a Gaussian distributed field only. The TAP magnetizations are
stable fixed points if an AT stability criterion is fulfilled. We illustrate
our method explicitly for coupling matrices drawn from the random orthogonal
ensemble.Comment: 27 pages, 6 Figures Published in Journal of Physics A: Mathematical
and Theoretical, Volume 49, Number 11, 201
S-AMP: Approximate Message Passing for General Matrix Ensembles
In this work we propose a novel iterative estimation algorithm for linear
observation systems called S-AMP whose fixed points are the stationary points
of the exact Gibbs free energy under a set of (first- and second-) moment
consistency constraints in the large system limit. S-AMP extends the
approximate message-passing (AMP) algorithm to general matrix ensembles. The
generalization is based on the S-transform (in free probability) of the
spectrum of the measurement matrix. Furthermore, we show that the optimality of
S-AMP follows directly from its design rather than from solving a separate
optimization problem as done for AMP.Comment: 5 pages, 1 figur
S-AMP for Non-linear Observation Models
Recently we extended Approximate message passing (AMP) algorithm to be able
to handle general invariant matrix ensembles. In this contribution we extend
our S-AMP approach to non-linear observation models. We obtain generalized AMP
(GAMP) algorithm as the special case when the measurement matrix has zero-mean
iid Gaussian entries. Our derivation is based upon 1) deriving expectation
propagation (EP) like algorithms from the stationary-points equations of the
Gibbs free energy under first- and second-moment constraints and 2) applying
additive free convolution in free probability theory to get low-complexity
updates for the second moment quantities.Comment: 6 page
Exact solution to the random sequential dynamics of a message passing algorithm
We analyze the random sequential dynamics of a message passing algorithm for
Ising models with random interactions in the large system limit. We derive
exact results for the two-time correlation functions and the speed of
convergence. The {\em de Almedia-Thouless} stability criterion of the static
problem is found to be necessary and sufficient for the global convergence of
the random sequential dynamics.Comment: Accepted for publication in Physical Review E Lette
Capacity Scaling in MIMO Systems with General Unitarily Invariant Random Matrices
We investigate the capacity scaling of MIMO systems with the system
dimensions. To that end, we quantify how the mutual information varies when the
number of antennas (at either the receiver or transmitter side) is altered. For
a system comprising receive and transmit antennas with , we find
the following: By removing as many receive antennas as needed to obtain a
square system (provided the channel matrices before and after the removal have
full rank) the maximum resulting loss of mutual information over all
signal-to-noise ratios (SNRs) depends only on , and the matrix of
left-singular vectors of the initial channel matrix, but not on its singular
values. In particular, if the latter matrix is Haar distributed the ergodic
rate loss is given by nats. Under
the same assumption, if with the ratio
fixed, the rate loss normalized by converges almost surely to
bits with denoting the binary entropy function. We also quantify and
study how the mutual information as a function of the system dimensions
deviates from the traditionally assumed linear growth in the minimum of the
system dimensions at high SNR.Comment: Accepted for publication in the IEEE Transactions on Information
Theor
Dynamical Functional Theory for Compressed Sensing
We introduce a theoretical approach for designing generalizations of the
approximate message passing (AMP) algorithm for compressed sensing which are
valid for large observation matrices that are drawn from an invariant random
matrix ensemble. By design, the fixed points of the algorithm obey the
Thouless-Anderson-Palmer (TAP) equations corresponding to the ensemble. Using a
dynamical functional approach we are able to derive an effective stochastic
process for the marginal statistics of a single component of the dynamics. This
allows us to design memory terms in the algorithm in such a way that the
resulting fields become Gaussian random variables allowing for an explicit
analysis. The asymptotic statistics of these fields are consistent with the
replica ansatz of the compressed sensing problem.Comment: 5 pages, accepted for ISIT 201
- …